Computing Lexical Cohesion as a Tool for Text Analysis

Recognizing coherent structure of a text is an essential task in natural language understanding. It is necessary, for example, to resolve anaphora, ellipsis, and ambiguity. One of the dominant factors of coherence of the text structure is lexical cohesion, namely the dependency relationship between words based on associative relations in common knowledge. This thesis proposes an objective and computationally feasible method for measuring lexical cohesion, especially semantic relations, between words. Lexical cohesion between words is computed on a semantic network constructed systematically from a subset of an ordinary English dictionary. Spreading activation on the semantic network analyses the meaning of a word into a 2,851-dimensional semantic space and computes the strength of lexical cohesion between any two words in the dictionary. As an evaluation of the measurement of lexical cohesion, this thesis then presents a quantitative indicator, Lexical Cohesion Pro le (LCP), for segmenting narratives into scenes, the smallest domain in which text coherence can be de ned. LCP is a record of the density of lexical cohesion of words in a window (of 51 words long, in an example) that moves forwards word by word on the text. Hills and valleys in a graph of LCP plotted against word position indicate alternation of scenes in the text. A psychological experiment shows that LCP correlates closely with the human judgements. The evaluation through the text-level application reveals that the proposed measurement of lexical cohesion works well as an indicator of coherent structure of a text. The measurement of lexical cohesion provides semantic information for text analysis. The segmentation scheme provides the frame work for recognizing coherent text structure. Both can be applied to various studies in a broad range of elds in natural language processing.